Encoding High-Cardinality String Categorical Variables

Statistical models usually require vector representations of categorical variables, using for instance one-hot encoding. This strategy breaks down when the number categories grows, as it creates high-dimensional feature vectors. Additionally, string entries, encoding does not capture information in their representation.Here, we seek low-dimensional high-cardinality variables. Ideally, these should be: scalable to many categories; interpretable end users; and facilitate statistical analysis. We introduce two approaches categories: a Gamma-Poisson matrix factorization on substring counts, min-hash encoder, fast approximation similarities. show that turns set inclusions into inequality relations are easier learn. Both streamable. Experiments real simulated data methods improve supervised learning with recommend following: if scalability is central, encoder best option any fit; interpretability important, alternative, can be interpreted inferred informative names. enable autoML original entries they remove need engineering or cleaning.